Calibration and the Evaluation of Predictive Learners

Calibration is the degree to which an agent's probability estimates (subjective probabilities) correspond to actual frequencies (or, the objective probabilities underlying them). Cognitive psychologists have studied human calibration as a part of their program to investigate how human cognition deviates from the ideal. In gambling, one attempts to estimate the odds of an event so as to maximize return. There are two major factors involved: knowledge of the domain and the meta-knowledge of the limits of that domain knowledge|i.e., calibration. Machine learning practice has emphasized measures which evaluate the amount of domain knowledge acquired, and which ignore calibration. Here I argue that gambling is an appropriate metaphor for the prediction task expected of machine learners, and that an information-theoretic measure of predictive ability, incorporating both domain knowledge and calibration, is an impartial and useful summative measure of learning ability.